This site is intended for Healthcare Professionals only

chatbot-summary
Analysis bookmark icon off

Weighing up the evidence: The ChatGPT will see you now

Getting to see a GP these days can be difficult, so could an artificial intelligence chatbot serve as a suitable alternative?

Many patients wait for what must seem like an eternity to get through to their GP practice, only to discover that there are no more appointments available that day.

To get a sense of the extent of the problem, data produced by NHS Digital shows that in May 2023, for example, only 44 per cent of appointments took place on the day that they were booked, 47 per cent of which were with a GP. 

However, with the number of full-time GPs having fallen from 28,590 in 2015 to 26,718 in 2023, and a recognition that primary care is under enormous pressure, could access be improved by replacing GPs with an AI chatbot? 

Impressive results

The Chat Generative Pretrained Transformer (GPT) represents an advanced chatbot AI system that was launched in November 2022 – although GPT-1 was first introduced back in 2018. The novelty of the system was an ability for unsupervised learning (i.e. it could basically ‘teach’ itself).

ChatGPT was able to pass the medical licensing exam in the US – but how well did it perform when responding to questions posed by patients? This was examined in a recent study published in JAMA Internal Medicine. Researchers compared the response to 195 questions given by ChatGPT and physicians. The results were impressive, favouring ChatGPT over physicians in 79 per cent of the 585 evaluations.

Equally remarkable was the fact that the proportion of responses rated as ‘good’ or ‘very good’ were 79 per cent for ChatGPT but only 22 per cent for physicians. The ChatGPT responses were deemed to be nearly 10 times more empathetic than those given by physicians. 

What is most compelling about this study is that it used GPT-3.5, an earlier version of the software, which has now been superseded by GPT-4. If the study was repeated, the results might be even more impressive.

How well ChatGPT compares with primary care physicians when faced with a specific clinical condition was examined in a study of mild and severe depression. Researchers found that both ChatGPT-3.5 and ChatGPT-4 provided recommendations aligned with clinical guidelines. However, unlike human physicians, the ChatGPT advice was not influenced by either gender or socio-economic status, which often affects a physician’s choice of treatment.

Despite these positive findings, other work suggests that ChatGPT may not yet be able to serve as a replacement for a GP. For example, successfully passing the applied knowledge test, which tests higher-order reasoning rather than recall of facts, is a requirement for membership of the Royal College of General Practitioners. When ChatGPT-3.5 took this test, its overall performance was 60.17 per cent – below the mean passing mark in the previous two years of 70.42 per cent. ChatGPT also expressed uncertainty in a small number of cases and provided an incorrect response. 

Issues with accuracy and bias

A recent systematic review explored the role of ChatGPT in the diagnosis and treatment of patients and its potential role in medical research.

In a review of 118 articles, it seemed that while ChatGPT could assist with patient enquiries, write medical notes, assist in decision-making and patient education, the solutions were often either insufficient or contradictory. The researchers concluded that ChatGPT could help with patient treatment and research but there remained issues with accuracy and bias. 

With algorithm or protocol-based healthcare seen as a way of reducing medical errors, AI technologies clearly have an important role to play. In fact, the use of an algorithm serves as an invaluable aid when teaching medical students about a classic or textbook patient – which probably explains why ChatGPT was able to pass medical exams – but in reality patients are more nuanced, with complex medical and social backgrounds, differing treatment goals, and a body language and tone of voice that cannot be accurately captured by ChatGPT. 

There are also other pertinent questions raised by relying solely on the output from ChatGPT. For instance, who retains overall liability if the system provides an incorrect diagnosis or treatment plan, which ultimately leads to patient harm? 

ChatGPT has numerous attributes that should be utilised; however, at the present time, it is unlikely to replace a GP. Nevertheless, in the future, ChatGPT could serve as an important member of the primary healthcare team, working alongside GPs and pharmacists to improve the efficiency of consultations but with an overarching aim of achieving the best possible patient outcomes.

Role in vaccine hesitancy and STIs?

New research presented at this year’s ESCMID Global Congress in Barcelona, Spain (April 27-30) suggests that ChatGPT could be an effective tool to help reduce vaccine hesitancy among the general public, as well as providing helpful advice on sexually transmitted infections (STIs).

Researchers from the National University Health System in Singapore tested ChatGPT to see whether it could successfully provide answers to common questions about vaccination and STIs. ChatGPT was asked 15 commonly asked questions on vaccine hesitancy and 17 on STIs.

“Our findings showed that ChatGPT displayed a remarkable ability to accurately respond to a wide breadth of commonly asked questions, encompassing topics such as misconceptions around mRNA vaccination and the importance of testing for STIs,” says lead author Dr Matthew Koh. “In the majority of instances, ChatGPT performed at the level of advice provided by professional organisations and guidelines.”

Overall, ChatGPT’s responses to vaccine hesitancy were accurate and may help individuals who have vaccine-related misconceptions, says Dr Koh. While it provided “good general advice about STIs, it lacked specificity and could not tailor its advice to [an] individual’s risk of acquiring STIs”, says Dr Koh.

“We believe that ChatGPT should be used in conjunction with, or in addition to, a physician consultation – the human response is still needed to contextualise the advice that is given to a specific individual’s situation and for it to make the most sense and be the most helpful for the patient.”

Copy Link copy link button

Analysis

It’s time to delve deeper into health discussions. Broaden your understanding of a range of pharmacy news and topics through in-depth analysis and insight.

Share: